Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid time-consuming irreproducible manual process trial-and-error to find well-performing hyperparameter configurations, various automatic optimization (HPO) methods—for example, based on resampling error estimation for supervised learning—can employed. After introducing HPO from general perspective, this paper reviews important methods, simple techniques such as grid or random search more advanced methods like evolution strategies, Bayesian optimization, Hyperband, racing. This work gives practical recommendations regarding choices made when conducting HPO, including the themselves, performance evaluation, how combine with pipelines, runtime improvements, parallelization. article is categorized under: Algorithmic Development > Statistics Technologies Machine Learning Prediction